Spatially-resolved dynamic dimming for augmented reality device

ABSTRACT

Techniques are described for operating an optical system. In some embodiments, light associated with a world object is received at the optical system. Virtual image light is projected onto an eyepiece of the optical system. A portion of a system field of view of the optical system to be at least partially dimmed is determined based on information detected by the optical system. A plurality of spatially-resolved dimming values for the portion of the system field of view may be determined based on the detected information. The detected information may include light information, gaze information, and/or image information. A dimmer of the optical system may be adjusted to reduce an intensity of light associated with the world object in the portion of the system field of view according to the plurality of dimming values.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. ProvisionalPatent Application No. 62/725,993, filed Aug. 31, 2018, entitled“SPATIALLY-RESOLVED DYNAMIC DIMMING FOR AUGMENTED REALITY DEVICE,” andU.S. Provisional Patent Application No. 62/858,252 filed Jun. 6, 2019,entitled “SPATIALLY-RESOLVED DYNAMIC DIMMING FOR AUGMENTED REALITYDEVICE,” the contents of which are herein incorporated in theirentirety.

BACKGROUND OF THE INVENTION

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” or “augmentedreality” experiences, wherein digitally reproduced images or portionsthereof are presented to a user in a manner wherein they seem to be, ormay be perceived as, real. A virtual reality, or “VR,” scenariotypically involves presentation of digital or virtual image informationwithout transparency to other actual real-world visual input; anaugmented reality, or “AR,” scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the actual world around the user.

Despite the progress made in these display technologies, there is a needin the art for improved methods, systems, and devices related toaugmented reality systems, particularly, display systems.

SUMMARY OF THE INVENTION

The present disclosure relates generally to techniques for improvingoptical systems in varying ambient light conditions. More particularly,embodiments of the present disclosure provide systems and methods foroperating an augmented reality (AR) device comprising a dimming element.Although the present invention is described in reference to an ARdevice, the disclosure is applicable to a variety of applications incomputer vision and image display systems.

A summary of the invention is provided below in reference to severalexamples. As used below, any reference to a series of examples is to beunderstood as a reference to each of those examples disjunctively (e.g.,“Examples 1-4” is to be understood as “Examples 1, 2, 3, or 4”).

Example 1 is a method of operating an optical system, the methodcomprising: receiving, at the optical system, light associated with aworld object; projecting virtual image light onto an eyepiece;determining a portion of a system field of view of the optical system tobe at least partially dimmed based on information detected by theoptical system; and adjusting a dimmer to reduce an intensity of thelight associated with the world object in the portion of the systemfield of view.

Example 2 is the method of example(s) 1, wherein the optical systemcomprises a light sensor configured to detect light informationcorresponding to the light associated with the world object, wherein thedetected information includes the light information.

Example 3 is the method of example(s) 1-2, wherein the light informationincludes a plurality of spatially-resolved light values.

Example 4 is the method of example(s) 1-3, wherein the light informationincludes a global light value.

Example 5 is the method of example(s) 1-4, wherein the optical systemcomprises an eye tracker configured to detect gaze informationcorresponding to an eye of a user of the optical system, wherein thedetected information includes the gaze information.

Example 6 is the method of example(s) 1-5, wherein the gaze informationincludes a pixel location that intersects with a gaze vector of the eyeof the user.

Example 7 is the method of example(s) 1-6, wherein the gaze informationincludes one or more of a pupil position of the eye of the user, acenter of rotation of the eye of the user, a pupil size of the eye ofthe user, a pupil diameter of the eye of the user, and cone and rodlocations of the eye of the user.

Example 8 is the method of example(s) 1-7, further comprising: detectingimage information corresponding to the virtual image light, wherein thedetected information includes the image information.

Example 9 is the method of example(s) 1-8, wherein the image informationincludes a plurality of spatially-resolved image brightness values.

Example 10 is the method of example(s) 1-9, wherein the imageinformation includes a global image brightness value.

Example 11 is the method of example(s) 1-10, further comprising:determining a plurality of spatially-resolved dimming values for theportion of the system field of view based on the detected information,wherein the dimmer is adjusted according to the plurality of dimmingvalues.

Example 12 is the method of example(s) 1-11, wherein the dimmercomprises a plurality of pixels.

Example 13 is the method of example(s) 1-12, wherein the dimmer isadjusted to completely block the intensity of the light associated withthe world object in all of the system field of view.

Example 14 is the method of example(s) 1-13, further comprising:adjusting a brightness associated with the virtual image light.

Example 15 is the method of example(s) 1-14, wherein the virtual imagelight is characterized by an image field of view, and wherein the imagefield of view is equal to the system field of view.

Example 16 is the method of example(s) 1-15, wherein determining aportion of a system field of view of the optical system to be at leastpartially dimmed is based at least partly on the at least one worldobject.

Example 17 is the method of example(s) 1-16, wherein determining aportion of a system field of view of the optical system to be at leastpartially dimmed is based at least partly on at least one objectincluded in the virtual image.

Example 18 is a non-transitory computer-readable medium comprisinginstructions that, when executed by a processor, cause the processor toperform operations comprising: receiving, at an optical system, lightassociated with a world object; projecting virtual image light onto aneyepiece; determining a portion of a system field of view of the opticalsystem to be at least partially dimmed based on information detected bythe optical system; and adjusting a dimmer to reduce an intensity of thelight associated with the world object in the portion of the systemfield of view.

Example 19 is the non-transitory computer-readable medium of example(s)18, wherein the optical system comprises a light sensor configured todetect light information corresponding to the light associated with theworld object, wherein the detected information includes the lightinformation.

Example 20 is the non-transitory computer-readable medium of example(s)19, wherein the light information includes a plurality ofspatially-resolved light values.

Example 21 is the non-transitory computer-readable medium of example(s)19, wherein the light information includes a global light value.

Example 22 is the non-transitory computer-readable medium of example(s)18, wherein the optical system comprises an eye tracker configured todetect gaze information corresponding to an eye of a user of the opticalsystem, wherein the detected information includes the gaze information.

Example 23 is the non-transitory computer-readable medium of example(s)22, wherein the gaze information includes a pixel location thatintersects with a gaze vector of the eye of the user.

Example 24 is the non-transitory computer-readable medium of example(s)22, wherein the gaze information includes one or more of a pupilposition of the eye of the user, a center of rotation of the eye of theuser, a pupil size of the eye of the user, a pupil diameter of the eyeof the user, and cone and rod locations of the eye of the user.

Example 25 is the non-transitory computer-readable medium of example(s)18, wherein the operations further comprise: detecting image informationcorresponding to the virtual image light, wherein the detectedinformation includes the image information.

Example 26 is the non-transitory computer-readable medium of example(s)25, wherein the image information includes a plurality ofspatially-resolved image brightness values.

Example 27 is the non-transitory computer-readable medium of example(s)25, wherein the image information includes a global image brightnessvalue.

Example 28 is the non-transitory computer-readable medium of example(s)18, wherein the operations further comprise: determining a plurality ofspatially-resolved dimming values for the portion of the system field ofview based on the detected information, wherein the dimmer is adjustedaccording to the plurality of dimming values.

Example 29 is the non-transitory computer-readable medium of example(s)18, wherein the dimmer comprises a plurality of pixels.

Example 30 is the non-transitory computer-readable medium of example(s)18, wherein the dimmer is adjusted to completely block the intensity ofthe light associated with the world object in all of the system field ofview.

Example 31 is the non-transitory computer-readable medium of example(s)18, wherein the operations further comprise: adjusting a brightnessassociated with the virtual image light.

Example 32 is the non-transitory computer-readable medium of example(s)18, wherein the virtual image light is characterized by an image fieldof view, and wherein the image field of view is equal to the systemfield of view.

Example 33 is an optical system comprising: a projector configured toproject virtual image light onto an eyepiece; a dimmer configured to dimlight associated with a world object; a processor communicativelycoupled to the projector and the dimmer, wherein the processor isconfigured to perform operations comprising: determining a portion of asystem field of view of the optical system to be at least partiallydimmed based on information detected by the optical system; andadjusting the dimmer to reduce an intensity of the light associated withthe world object in the portion of the system field of view.

Example 34 is the optical system of example(s) 33, further comprising: alight sensor configured to detect light information corresponding to thelight associated with the world object, wherein the detected informationincludes the light information.

Example 35 is the optical system of example(s) 34, wherein the lightinformation includes a plurality of spatially-resolved light values.

Example 36 is the optical system of example(s) 34, wherein the lightinformation includes a global light value.

Example 37 is the optical system of example(s) 33, further comprising:an eye tracker configured to detect gaze information corresponding to aneye of a user of the optical system, wherein the detected informationincludes the gaze information.

Example 38 is the optical system of example(s) 37, wherein the gazeinformation includes a pixel location that intersects with a gaze vectorof the eye of the user.

Example 39 is the optical system of example(s) 37, wherein the gazeinformation includes one or more of a pupil position of the eye of theuser, a center of rotation of the eye of the user, a pupil size of theeye of the user, a pupil diameter of the eye of the user, and cone androd locations of the eye of the user.

Example 40 is the optical system of example(s) 33, wherein theoperations further comprise: detecting image information correspondingto the virtual image light, wherein the detected information includesthe image information.

Example 41 is the optical system of example(s) 40, wherein the imageinformation includes a plurality of spatially-resolved image brightnessvalues.

Example 42 is the optical system of example(s) 40, wherein the imageinformation includes a global image brightness value.

Example 43 is the optical system of example(s) 33, wherein theoperations further comprise: determining a plurality ofspatially-resolved dimming values for the portion of the system field ofview based on the detected information, wherein the dimmer is adjustedaccording to the plurality of dimming values.

Example 44 is the optical system of example(s) 33, wherein the dimmercomprises a plurality of pixels.

Example 45 is the optical system of example(s) 33, wherein the dimmer isadjusted to completely block the intensity of the light associated withthe world object in all of the system field of view.

Example 46 is the optical system of example(s) 33, wherein theoperations further comprise: adjusting a brightness associated with thevirtual image light.

Example 47 is the optical system of example(s) 33, wherein the virtualimage light is characterized by an image field of view, and wherein theimage field of view is equal to the system field of view.

Example 48 is an optical system comprising: a frame configured to beworn about a head of a user of the optical system; a dimming componentcarried by the frame and configured to be positioned between an eye ofthe user and an environment of the user; an eye tracker configured tomonitor a positioning of the eye of the user; and control circuitrycommunicatively coupled to the dimming component and the eye tracker,the control circuitry configured to: receive data from the eye tracker;determine, based on the data received from the eye tracker, a locationalong an optical axis of the eye of the user at which a particularanatomical region of the eye of the user is positioned; identify one ormore points in three-dimensional space located within the environment ofthe user; and for each of the one or more identified points within theenvironment of the user: identify a set of one or more pixels of thedimming component based at least in part on the determined location ofthe particular anatomical region of the eye of the user and therespective point in three-dimensional space located within theenvironment of the user; and control the dimming component to dim theidentified set of one or more pixels.

Example 49 is the optical system of example(s) 48, wherein theparticular anatomical region of the eye of the user comprises a centerof rotation of the eye of the user.

Example 50 is the optical system of example(s) 48, wherein theparticular anatomical region of the eye of the user comprises a centerof a pupil of the eye of the user.

Example 51 is the optical system of example(s) 48, further comprising: aprojector configured to emit light representing virtual content; and awaveguide carried by the frame and configured to be positioned betweenthe eye of the user and the dimming component, wherein the waveguide isconfigured to receive and direct light from the projector to the eye ofthe user.

Example 52 is the optical system of example(s) 51, wherein the controlcircuitry is communicatively coupled to the projector, the controlcircuitry further configured to control the projector to emit lightrepresenting one or more pixels of virtual content.

Example 53 is the optical system of example(s) 52, wherein the one ormore points in three-dimensional space located within the environment ofthe user correspond to one or more locations in three-dimensional spaceat which the one or more pixels of virtual content are to be perceivedby the user, respectively.

Example 54 is the optical system of example(s) 52, wherein the one ormore pixels of virtual content comprise a plurality of pixels of avirtual object.

Example 55 is the optical system of example(s) 54, wherein the one ormore points in three-dimensional space located within the environment ofthe user correspond to one or more locations in three-dimensional spaceat which one or more pixels of a virtual shadow associated with thevirtual object is to be perceived by the user, respectively.

Example 56 is the optical system of example(s) 48, wherein the one ormore points in three-dimensional space located within the environment ofthe user correspond to one or more points in three-dimensional spacephysically occupied by a real world object in the environment of theuser.

Example 57 is the optical system of example(s) 48, wherein to identifythe set of one or more pixels of the dimming component, the controlcircuitry is configured to: cast a set of one or more rays from therespective point in three-dimensional space located within theenvironment of the user to the determined location of the particularanatomical region of the eye of the user; and identify a set of one ormore points of intersection between the set of one or more rays and thedimming component.

Example 58 is the optical system of example(s) 48, wherein the dimmingcomponent is curved in shape.

Example 59 is the optical system of example(s) 48, wherein the controlcircuitry is further configured to determine a set of one or moredimming values for the identified set of one or more pixels of thedimming component, respectively, and wherein the control circuitry isconfigured to control the dimming component to dim the identified set ofone or more pixels in accordance with the determined set of one or moredimming values.

Example 60 is the optical system of example(s) 59, wherein the controlcircuitry is further configured to determine one or more characteristicsof the eye of the user based on the data received from the eye tracker,and wherein the control circuitry is configured to determine the set ofone or more dimming values for the identified set of one or more pixelsof the dimming component, respectively, based at least in part on theone or more determined characteristics of the eye of the user.

Example 61 is the optical system of example(s) 60, wherein the one ormore characteristics of the eye of the user include one or more of apupil size of the eye of the user, a pupil diameter of the eye of theuser, cone and rod locations of the eye of the user, and anaccommodative state of a lens of the eye of the user.

Example 62 is the optical system of example(s) 61, wherein the controlcircuitry is configured to identify the set of one or more pixels of thedimming component based at least in part on the one or more determinedcharacteristics of the eye of the user.

Example 63 is the optical system of example(s) 59 further comprising: aprojector communicatively coupled to the control circuitry andconfigured to emit light representing virtual content; and a waveguidecarried by the frame and configured to be positioned between the eye ofthe user and the dimming component, wherein the waveguide is configuredto receive and direct light from the projector to the eye of the user,wherein the control circuitry is further configured to control theprojector to emit light representing one or more pixels of virtualcontent at one or more levels of brightness, respectively, and whereinthe control circuitry is configured to determine the set of one or moredimming values for the identified set of one or more pixels of thedimming component, respectively, based at least in part on the one ormore levels of brightness of the one or more pixels of virtual content.

Example 64 is the optical system of example(s) 63, wherein the controlcircuitry is configured to determine the set of one or more dimmingvalues for the identified set of one or more pixels of the dimmingcomponent, respectively, based at least in part on one or more of apredetermined contrast and a predetermined level of visibility specifiedfor the virtual content.

Example 65 is the optical system of example(s) 63, wherein the virtualcontent comprises a virtual object, and wherein the control circuitry isconfigured to identify the set of one or more pixels of the dimmingcomponent based at least in part on one or more characteristics of thevirtual object.

Example 66 is the optical system of example(s) 65, wherein the one ormore characteristics of the virtual object include one or more of a sizeof the virtual object, a shape of the virtual object, a position in theenvironment of the user at which the virtual object is to be perceivedby the user, and a depth at which the virtual object is to be perceivedby the user.

Example 67 is the optical system of example(s) 59 further comprising: anoptical sensor communicatively coupled to the control circuitry andconfigured to monitor one or more levels of brightness of lightassociated with one or more portions of the environment of the user,respectively, and wherein the control circuitry is configured todetermine the set of one or more dimming values for the identified setof one or more pixels of the dimming component, respectively, based atleast in part on the one or more levels of brightness associated withthe one or more portions of the environment of the user.

Example 68 is the optical system of example(s) 67, wherein the opticalsensor comprises a camera.

Example 69 is the optical system of example(s) 67, wherein the opticalsensor comprises one or more photodiodes.

Example 70 is the optical system of example(s) 67 further comprising: aprojector communicatively coupled to the control circuitry andconfigured to emit light representing virtual content; and a waveguidecarried by the frame and configured to be positioned between the eye ofthe user and the dimming component, wherein the waveguide is configuredto receive and direct light from the projector to the eye of the user,and wherein the control circuitry is further configured to control theprojector to emit light representing one or more pixels of virtualcontent.

Example 71 is the optical system of example(s) 70, wherein the virtualcontent comprises a virtual object, and wherein the one or more portionsof the environment of the user with which the one or more levels ofbrightness are associated include a particular portion of theenvironment of the user that is to be perceived by the user as occludedby the virtual object.

Example 72 is the optical system of example(s) 70, wherein the controlcircuitry is further configured to control the projector to emit lightrepresenting one or more pixels of virtual content based at least inpart on the one or more levels of brightness associated with the one ormore portions of the environment of the user.

Example 73 is the optical system of example(s) 48, wherein the eyetracker is configured to monitor the positioning of the eye of the userrelative to the dimming component.

Example 74 is an optical system comprising: a frame configured to beworn about a head of a user of the optical system; a left dimmingcomponent carried by the frame and configured to be positioned between aleft eye of the user and an environment of the user; a right dimmingcomponent carried by the frame and configured to be positioned between aright eye of the user and the environment of the user; and controlcircuitry communicatively coupled to the left and right dimmingcomponents, the control circuitry configured to: identify one or morepoints in three-dimensional space located within the environment of theuser; and for each of the one or more identified points within theenvironment of the user: identify a set of one or more pixels of theleft dimming component based at least in part on the respective point inthree-dimensional space located within the environment of the user;identify a set of one or more pixels of the right dimming componentbased at least in part on the respective point in three-dimensionalspace located within the environment of the user; control the leftdimming component to dim the identified set of one or more pixels of theleft dimming component; and control the right dimming component to dimthe identified set of one or more pixels of the right dimming component.

Example 75 is the optical system of example(s) 74 further comprising: aleft eye tracker communicatively coupled to the control circuitry andconfigured to monitor a positioning of the left eye of the user; and aright eye tracker communicatively coupled to the control circuitry andconfigured to monitor a positioning of the left eye of the user; whereinthe control circuitry is further configured to: receive data from theleft and right eye trackers; determine, based on the data received fromthe left eye tracker, a location along an optical axis of the left eyeof the user at which a particular anatomical region of the left eye ofthe user is positioned; and determine, based on the data received fromthe right eye tracker, a location along an optical axis of the right eyeof the user at which a particular anatomical region of the right eye ofthe user is positioned;

Example 76 is the optical system of example(s) 75, wherein the controlcircuitry is configured to: identify the set of one or more pixels ofthe left dimming component based at least in part on the determinedlocation of the particular anatomical region of the left eye of the userand the respective point in three-dimensional space located within theenvironment of the user; and identify the set of one or more pixels ofthe right dimming component based at least in part on the determinedlocation of the particular anatomical region of the right eye of theuser and the respective point in three-dimensional space located withinthe environment of the user.

Numerous benefits are achieved by way of the present disclosure overconventional techniques. For example, augmented reality (AR) devicesdescribed herein may be used in varying light levels, from dark indoorsto bright outdoors, by globally dimming and/or selectively dimming theambient light reaching the user's eyes. Embodiments of the presentinvention allow for AR and virtual reality (VR) capabilities in a singledevice by using the pixelated dimmer to attenuate the world light bygreater than 99%. Embodiments of the present invention also mitigatevergence accommodation conflict using a variable focal element withdiscrete or continuous variable depth plane switching technologies.Embodiments of the present invention improve the battery life of the ARdevice by optimizing the projector brightness based on the amount ofdetected ambient light. Other benefits of the present disclosure will bereadily apparent to those skilled in the art.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an augmented reality (AR) scene as viewed through awearable AR device according to some embodiments described herein.

FIG. 2A illustrates one or more general features of an AR deviceaccording to the present invention.

FIG. 2B illustrates an example of an AR device in which a dimmed area isdetermined based on detected light information.

FIG. 2C illustrates an example of an AR device in which a dimmed area isdetermined based on a virtual image.

FIG. 2D illustrates an example of an AR device in which a dimmed area isdetermined based on gaze information.

FIG. 3 illustrates a schematic view of a wearable AR device according tothe present invention.

FIG. 4 illustrates a method for operating an optical system.

FIG. 5 illustrates an AR device with an eyepiece and a pixelated dimmingelement.

FIG. 6 illustrates a technique for determining a gaze vector based on apupil position of an eye of a user.

FIG. 7 illustrates a technique for determining a gaze vector based on acenter of rotation of an eye of a user.

FIG. 8 illustrates a technique for determining a gaze vector based ondetected light information and cone and rod locations within an eye.

FIG. 9 illustrates a determined gaze vector in high light conditions inwhich a pupil is contracted.

FIG. 10 illustrates determined gaze vectors in low light conditions inwhich a pupil is dilated.

FIG. 11 illustrates three techniques for determining a gaze vector inhigh light conditions and corresponding dimmed areas.

FIG. 12 illustrates three techniques for determining a gaze vector inlow light conditions and corresponding dimmed areas.

FIG. 13 illustrates a dimmer having been adjusted to produce a dimmedarea determined using a gaze vector calculated using a pupil position.

FIG. 14 illustrates a dimmer having been adjusted to produce a dimmedarea determined using a gaze vector calculated using cone and rodlocations in high light conditions.

FIG. 15 illustrates a dimmer having been adjusted to produce a dimmedarea determined using a gaze vector calculated using cone and rodlocations in low light conditions.

FIG. 16 illustrates an example in which a dimmed area includes a centerportion within an annular region.

FIG. 17 illustrates a dimmer having been adjusted to produce a dimmedarea determined using a gaze vector calculated using a center ofrotation of an eye.

FIGS. 18A and 18B illustrate an approach for determining a portion ofthe system field of view to be dimmed based on image information.

FIGS. 19A and 19B illustrate an approach for determining a portion ofthe system field of view to be dimmed based on image information.

FIG. 20 illustrates an example of improving the opacity of the virtualcontent by adjusting the dimmer and/or adjusting the projector.

FIG. 21 illustrates an example of improving the opacity of the virtualcontent by dimming a portion of the system field of view correspondingto the virtual object.

FIG. 22 illustrates a plot showing the relationship between virtualimage brightness and ambient light levels.

FIGS. 23A and 23B illustrate diagrams showing the effect of a smallocclusion on a world scene.

FIG. 24 illustrates a plot showing the effect of varying the occlusordiameter on the transmission of the dimming element as a function ofangular extent.

FIG. 25 illustrates an example of dimming using a single occlusor.

FIG. 26 illustrates an example of an architecture of an optical seethrough (OST) head-mounted display (HMD).

FIG. 27 illustrates an additional example of an architecture of anOST-HMD.

FIG. 28 illustrates an additional example of an architecture of anOST-HMD.

FIG. 29 illustrates a schematic view of an AR device according to thepresent invention.

FIG. 30 illustrates a method for sharpening out-of-focus pixelateddimming.

FIG. 31 illustrates a method for sharpening out-of-focus pixelateddimming.

FIG. 32 illustrates a method for sharpening out-of-focus pixelateddimming.

FIG. 33 illustrates a simplified computer system according to someembodiments described herein.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

An ongoing technical challenge with optical see through (OST) augmentedreality (AR) devices is the variation in the opacity and/or visibilityof the virtual content under varying ambient light conditions. Theproblem worsens in extreme lighting conditions such as a completely darkroom or outside in full bright sunlight. Embodiments of the presentinvention solve these and other problems by dimming the world light atdifferent spatial locations within the field of view of the AR device.The portion of the field of view to which dimming is applied and theamount of dimming that is applied are each determined based on variousinformation detected by the AR device. This information may includedetected ambient light, detected gaze information, and/or the detectedbrightness of the virtual content being projected. The functionality ofthe AR device is further improved by detecting a direction associatedwith the ambient light by, for example, detecting a plurality ofspatially-resolved light values. This allows the AR device to improveits battery life by only dimming the portions of the field of view inwhich dimming is needed and/or increasing the projector brightness incertain portions of the field of view. Accordingly, embodiments of thepresent invention enable usage of the AR device in a much wider varietyof conditions than traditionally possible.

FIG. 1 illustrates an AR scene 100 as viewed through a wearable ARdevice, according to some embodiments of the present invention. AR scene100 is depicted wherein a user of an AR technology sees a real-worldpark-like setting 106 featuring various real-world objects 130 such aspeople, trees, buildings in the background, and a real-world concreteplatform 120. In addition to these items, the user of the AR technologyalso perceives that they “see” various virtual objects 102 such as arobot statue 102-2 standing upon the real-world concrete platform 120,and a cartoon-like avatar character 102-1 flying by, which seems to be apersonification of a bumble bee, even though these elements (character102-1 and statue 102-2) do not exist in the real world. Due to theextreme complexity of the human visual perception and nervous system, itis challenging to produce a virtual reality (VR) or AR technology thatfacilitates a comfortable, natural-feeling, rich presentation of virtualimage elements amongst other virtual or real-world imagery elements.

FIG. 2A illustrates one or more general features of an AR device 200according to the present invention. In some embodiments, an AR device200 may include an eyepiece 202 and a dynamic dimmer 203 configured tobe transparent or semi-transparent when AR device 200 is in an inactivemode or an off mode such that a user may view one or more world objects230 when looking through eyepiece 202 and dynamic dimmer 203. Asillustrated, eyepiece 202 and dynamic dimmer 203 may be arranged in aside-by-side configuration and may form a system field of view that auser sees when looking through eyepiece 202 and dynamic dimmer 203. Insome embodiments, the system field of view is defined as the entiretwo-dimensional region occupied by one or both of eyepiece 202 anddynamic dimmer 203. Although FIG. 2A illustrates a single eyepiece 202and a single dynamic dimmer 203 (for illustrative reasons), AR device200 may include two eyepieces and two dynamic dimmers, one for each eyeof a user.

During operation, dynamic dimmer 203 may be adjusted to reduce anintensity of a world light 232 associated with world objects 230impinging on dynamic dimmer 203, thereby producing a dimmed area 236within the system field of view. Dimmed area 236 may be a portion orsubset of the system field of view, and may be partially or completelydimmed. Dynamic dimmer 203 may be adjusted according to a plurality ofspatially-resolved dimming values for dimmed area 236. Furthermore,during operation of AR device 200, projector 214 may project a virtualimage light 222 (i.e., light associated with virtual content) ontoeyepiece 202 which may be observed by the user along with world light232.

Projecting virtual image light 222 onto eyepiece 202 may cause a lightfield (i.e., an angular representation of virtual content) to beprojected onto the user's retina in a manner such that the userperceives the corresponding virtual content as being positioned at somelocation within the user's environment. For example, virtual image light222 outcoupled by eyepiece 202 may cause the user to perceive character202-1 as being positioned at a first virtual depth plane 210-1 andstatue 202-2 as being positioned at a second virtual depth plane 210-2.The user perceives the virtual content along with world light 232corresponding to one or more world objects 230, such as platform 120.

In some embodiments, AR device 200 may include an ambient light sensor234 configured to detect world light 232. Ambient light sensor 234 maybe positioned such that world light 232 detected by ambient light sensor234 is similar to and/or representative of world light 232 that impingeson dynamic dimmer 203 and/or eyepiece 202. In some embodiments, ambientlight sensor 234 may be configured to detect a plurality ofspatially-resolved light values corresponding to different pixels ofdynamic dimmer 203. In these embodiments, ambient light sensor 234 may,for example, correspond to an imaging sensor (e.g., CMOS, CCD, etc.) ora plurality of photodiodes (e.g., in an array or anotherspatially-distributed arrangement). In some embodiments, or in the sameembodiments, ambient light sensor 234 may be configured to detect aglobal light value corresponding to an average light intensity or asingle light intensity of world light 232. In these embodiments, ambientlight sensor 234 may, for example, correspond to a set of one or morephotodiodes. Other possibilities are contemplated.

FIG. 2B illustrates an example of AR device 200 in which dimmed area 236is determined based on detected light information corresponding to worldlight 232. Specifically, ambient light sensor 234 may detect world light232 associated with the sun and may further detect a direction and/or aportion of the system field of view at which world light 232 associatedwith the sun passes through AR device 200. In response, dynamic dimmer203 may be adjusted to set dimmed area 236 to cover a portion of thesystem field of view corresponding to the detected world light. Asillustrated, dynamic dimmer 203 may be adjusted so as to reduce theintensity of world light 232 at the center of dimmed area 236 at agreater amount than the extremities of dimmed area 236.

FIG. 2C illustrates an example of AR device 200 in which dimmed area 236is determined based on virtual image light 222. Specifically, dimmedarea 236 may be determined based on the virtual content perceived by theuser resulting from the user observing virtual image light 222. In someembodiments, AR device 200 may detect image information that includes alocation of virtual image light 222 (e.g., a location within dynamicdimmer 203 through which the user perceives the virtual content) and/ora brightness of virtual image light 222 (e.g., a brightness of theperceived virtual content and/or the light generated at projector 214),among other possibilities. As illustrated, dynamic dimmer 203 may beadjusted to set dimmed area 236 to cover a portion of the system fieldof view corresponding to virtual image light 222 or, alternatively, insome embodiments dimmed area 236 may cover a portion of the system fieldof view that is not aligned with virtual image light 222. In someembodiments, the dimming values of dimmed area 236 may be determinedbased on world light 232 detected by ambient light sensor 234 and/or thebrightness of virtual image light 222.

FIG. 2D illustrates an example of AR device 200 in which dimmed area 236is determined based on gaze information corresponding to an eye of auser. In some embodiments, the gaze information includes a gaze vector238 of the user and/or a pixel location of dynamic dimmer 203 at whichgaze vector 238 intersects with dynamic dimmer 203. As illustrated,dynamic dimmer 203 may be adjusted to set dimmed area 236 to cover aportion of the system field of view corresponding to an intersectionpoint (or intersection region) between gaze vector 238 and dynamicdimmer 203 or, alternatively, in some embodiments dimmed area 236 maycover a portion of the system field of view that does not correspond tothe intersection point (or intersection region) between gaze vector 238and dynamic dimmer 203. In some embodiments, the dimming values ofdimmed area 236 may be determined based on world light 232 detected byambient light sensor 234 and/or the brightness of virtual image light222. In some embodiments, gaze information may be detected by an eyetracker 240 mounted to AR device 200.

FIG. 3 illustrates a schematic view of a wearable AR device 300according to the present invention. AR device 300 may include a lefteyepiece 302A and a left dynamic dimmer 303A arranged in a side-by-sideconfiguration and a right eyepiece 302B and a right dynamic dimmer 303Balso arranged in a side-by-side configuration. In some embodiments, ARdevice 300 includes one or more sensors including, but not limited to: aleft front-facing world camera 306A attached directly to or near lefteyepiece 302A, a right front-facing world camera 306B attached directlyto or near right eyepiece 302B, a left side-facing world camera 306Cattached directly to or near left eyepiece 302A, a right side-facingworld camera 306D attached directly to or near right eyepiece 302B, aleft eye tracker 340A positioned so as to observe a left eye of a user,a right eye tracker 340B positioned so as to observe a right eye of auser, and an ambient light sensor 334. In some embodiments, AR device300 includes one or more image projection devices such as a leftprojector 314A optically linked to left eyepiece 302A and a rightprojector 314B optically linked to right eyepiece 302B.

Some or all of the components of AR device 300 may be head mounted suchthat projected images may be viewed by a user. In one particularimplementation, all of the components of AR device 300 shown in FIG. 3are mounted onto a single device (e.g., a single headset) wearable by auser. In another implementation, a processing module 350 is physicallyseparate from and communicatively coupled to the other components of ARdevice 300 by one or more wired and/or wireless connections. Forexample, processing module 350 may be mounted in a variety ofconfigurations, such as fixedly attached to a frame, fixedly attached toa helmet or hat worn by a user, embedded in headphones, or otherwiseremovably attached to a user (e.g., in a backpack-style configuration,in a belt-coupling style configuration, etc.).

Processing module 350 may include a processor 352 and an associateddigital memory 356, such as non-volatile memory (e.g., flash memory),both of which may be utilized to assist in the processing, caching, andstorage of data. The data may include data captured from sensors (whichmay be, e.g., operatively coupled to AR device 300) or otherwiseattached to a user, such as cameras 306, ambient light sensor 334, eyetrackers 340, microphones, inertial measurement units, accelerometers,compasses, GPS units, radio devices, and/or gyros. For example,processing module 350 may receive image(s) 320 from cameras 306.Specifically, processing module 350 may receive left front image(s) 320Afrom left front-facing world camera 306A, right front image(s) 320B fromright front-facing world camera 306B, left side image(s) 320C from leftside-facing world camera 306C, and right side image(s) 320D from rightside-facing world camera 306D. In some embodiments, image(s) 320 mayinclude a single image, a pair of images, a video comprising a stream ofimages, a video comprising a stream of paired images, and the like.Image(s) 320 may be periodically generated and sent to processing module350 while AR device 300 is powered on, or may be generated in responseto an instruction sent by processing module 350 to one or more of thecameras. As another example, processing module 350 may receive lightinformation from ambient light sensor 334. In some embodiments, some orall of the functionality of ambient light sensor 334 may be provided byway of one or more of world cameras 306A-306D. As another example,processing module 350 may receive gaze information from one or both ofeye trackers 340. As another example, processing module 350 may receiveimage information (e.g., image brightness values) from one or both ofprojectors 314.

Eyepieces 302A and 302B may comprise transparent or semi-transparentwaveguides configured to direct light from projectors 314A and 314B,respectively. Specifically, processing module 350 may cause leftprojector 314A to output a left virtual image light 322A onto lefteyepiece 302A (causing a corresponding light field associated with leftvirtual image light 322A to be projected onto the user's retina), andmay cause right projector 314B to output a right virtual image light322B onto right eyepiece 302B (causing a corresponding light fieldassociated with right virtual image light 322B to be projected onto theuser's retina). In some embodiments, each of eyepieces 302 may comprisea plurality of waveguides corresponding to different colors and/ordifferent depth planes. In some embodiments, dynamic dimmers 303 may becoupled to and/or integrated with eyepieces 302. For example, one ofdynamic dimmers 303 may be incorporated into a multi-layer eyepiece andmay form one or more layers that make up one of eyepieces 302.

Cameras 306A and 306B may be positioned to capture images thatsubstantially overlap with the field of view of a user's left and righteyes, respectively. Accordingly, placement of cameras 306 may be near auser's eyes but not so near as to obscure the user's field of view.Alternatively or additionally, cameras 306A and 306B may be positionedso as to align with the incoupling locations of virtual image light 322Aand 322B, respectively. Cameras 306C and 306D may be positioned tocapture images to the side of a user, e.g., in a user's peripheralvision or outside the user's peripheral vision. Image(s) 320C and 320Dcaptured using cameras 306C and 306D need not necessarily overlap withimage(s) 320A and 320B captured using cameras 306A and 306B.

One or more components of AR device 300 may be similar to one or morecomponents described in reference to FIGS. 2A-2D. For example, in thesome embodiments the functionality of eyepieces 302, dynamic dimmers303, projectors 314, ambient light sensor 334, and eye trackers 340 maybe similar to eyepiece 202, dynamic dimmer 203, projector 214, ambientlight sensor 234, and eye tracker 240, respectively. In someembodiments, the functionality of processing module 350 may beimplemented by two or more sets of electronic hardware components thatare housed separately but communicatively coupled. For example, thefunctionality of processing module 350 may be carried out by electronichardware components housed within a headset in conjunction withelectronic hardware components housed within a computing devicephysically tethered to the headset, one or more electronic deviceswithin the environment of the headset (e.g., smart phones, computers,peripheral devices, smart appliances, etc.), one or moreremotely-located computing devices (e.g., servers, cloud computingdevices, etc.), or a combination thereof. One example of such aconfiguration is described in further detail below in reference to FIG.29.

FIG. 4 illustrates a method 400 for operating an optical system (e.g.,AR device 200 or 300). Steps of method 400 may be performed in adifferent order than that shown in FIG. 4, and not all of the steps needbe performed. For example, in some embodiments, one or more of steps406, 408, and 410 may be omitted during performance of method 400. Oneor more steps of method 400 may be performed by a processor (e.g.,processor 352) or by some other component within the optical system.

At step 402, light (e.g., world light 232) associated with a worldobject (e.g., world objects 230) is received at the optical system. Theworld object may be any number of real-world objects, such as a tree, aperson, a house, a building, the sun, etc., that is viewed by a user ofthe optical system. In some embodiments, the light associated with theworld object is first received by a dynamic dimmer (e.g., dynamic dimmer203 or 303) or by an external cosmetic lens of the optical system. Insome embodiments, the light associated with the world object isconsidered to be received at the optical system when the light reachesone or more components of the optical system (e.g., when the lightreaches the dynamic dimmer).

At step 404, virtual image light (e.g., virtual image light 222 or 322)is projected onto an eyepiece (e.g., eyepiece 202 or 302). The virtualimage light may be projected onto the eyepiece by a projector (e.g.,projector 214 or 314) of the optical system. The virtual image light maycorrespond to a single image, a pair of images, a video comprising astream of images, a video comprising a stream of paired images, and thelike. In some embodiments, the virtual image light is considered to beprojected onto the eyepiece when any light associated with the virtualimage light reaches the eyepiece. In some embodiments, projecting thevirtual image light onto the eyepiece causes a light field (i.e., anangular representation of virtual content) to be projected onto theuser's retina in a manner such that the user perceives the correspondingvirtual content as being positioned at some location within the user'senvironment.

During steps 406, 408, and 410, information may be detected by theoptical system using, for example, one or more sensors of the opticalsystem. At step 406, light information corresponding to the lightassociated with the world object is detected. The light information maybe detected using a light sensor (e.g., ambient light sensor 234 or 334)mounted to the optical system. In some embodiments, the lightinformation includes a plurality of spatially-resolved light values.Each of the plurality of spatially-resolved light values may correspondto a two-dimensional position within the system field of view. Forexample, each of the light values may be associated with a pixel of thedynamic dimmer. In other embodiments, or in the same embodiments, thelight information may include a global light value. The global lightvalue may be associated with the entire system field of view (e.g., anaverage light value of light impinging on all pixels of the dynamicdimmer).

At step 408, gaze information corresponding to an eye of a user of theoptical system is detected. The gaze information may be detected usingan eye tracker (e.g., eye tracker 240 or 340) mounted to the opticalsystem. In some embodiments, the gaze information includes a gaze vector(e.g., gaze vector 238) of the eye of the user. In some embodiments, thegaze information includes one or more of a pupil position of the eye ofthe user, a center of rotation of the eye of the user, a pupil size ofthe eye of the user, a pupil diameter of the eye of the user, and coneand rod locations of the eye of the user. The gaze vector may bedetermined based on one or more components of the gaze information, suchas the pupil position, the center of rotation of the eye, the pupilsize, the pupil diameter, and/or the cone and rod locations. When thegaze vector is determined based on the cone and rod locations, it mayfurther be determined based on the light information (e.g., the globallight value) so as to determine an origin of the gaze vector within aretinal layer of the eye containing the cone and rod locations. In someembodiments, the gaze information includes a pixel or group of pixels ofthe dynamic dimmer at which the gaze vector intersects with the dynamicdimmer.

At step 410, image information corresponding to the virtual image light(e.g., virtual image light 222 or 322) projected by the projector ontothe eyepiece is detected. The image information may be detected by theprojector, by a processor (e.g., processor 352), or by a separate lightsensor. In some embodiments, the image information includes one or morelocations within the dynamic dimmer through which the user perceives thevirtual content when the user observes the virtual image light. In someembodiments, the image information includes a plurality ofspatially-resolved image brightness values (e.g., brightness of theperceived virtual content). For example, each of the image brightnessvalues may be associated with a pixel of the eyepiece or of the dynamicdimmer. In one particular implementation, when the processor sendsinstructions to the projector to project the virtual image light ontothe eyepiece, the processor may determine, based on the instructions,the spatially-resolved image brightness values. In another particularimplementation, when the projector receives the instructions from theprocessor to project the virtual image light onto the eyepiece, theprojector sends the spatially-resolved image brightness values to theprocessor. In another particular implementation, a light sensorpositioned on or near the eyepiece detects and sends thespatially-resolved image brightness values to the processor. In otherembodiments, or in the same embodiments, the image information includesa global image brightness value. The global image brightness value maybe associated with the entire system field of view (e.g., an averageimage brightness value of all of the virtual image light).

At step 412, a portion of the system field of view to be at leastpartially dimmed is determined based on the detected information. Thedetected information may include the light information detected duringstep 406, the gaze information detected during step 408, and/or theimage information detected during step 410. In some embodiments, theportion of the system field of view is equal to the entire system fieldof view. In various embodiments, the portion of the system field of viewmay be equal to 1%, 5%, 10%, 25%, 50%, or 75%, etc., of the system fieldof view. In some embodiments, the different types of information may beweighted differently in determining the portion to be at least partiallydimmed. For example, gaze information, when available, may be weightedmore heavily in determining the portion to be at least partially dimmedthan light information and image information. In one particularimplementation, each type of information may independently be used todetermine a different portion of the system field of view to be at leastpartially dimmed, and subsequently the different portions may becombined into a single portion using an AND or an OR operation.

In some embodiments, the information used to determine a portion of thesystem field of view to be at least partially dimmed includesinformation associated with one or more objects that are presentedwithin the virtual content. For example, the virtual content may includetext, navigational indicators (e.g., arrows), and/or other content. Theportion of the field of view in which such content is to be presented,and/or the field of view proximal to the content, can be dimmed suchthat the user can more easily read perceive and understand the content,and distinguish the content from world object(s). The dimmer canselectively dim one or more pixels and/or zone(s) of pixels, or enhanceviewing of the content. In one example, a section of the lower portionof the field of view can be selectively and dynamically dimmed to makeis easier for the user to see directional (e.g., navigation) arrows,text messages, and so forth. Such dimming may be performed while thecontent is being displayed in response to a determination that suchcontent is to be displayed, and the dimming may be removed when thecontent is no longer displayed. In some instances, the dimming may beperformed to mitigate artifacts caused by the pixel structure thatenables dimming over the entire field of view.

At step 414, a plurality of spatially-resolved dimming values for theportion of the system field of view are determined based on the detectedinformation. In some embodiments, the dimming values are determinedusing a formulaic approach based on a desired opacity or visibility ofthe virtual content. In one particular implementation, the visibility ofthe virtual content may be calculated using the following equation:

$V = \frac{I_{\max}\left( {1 - \frac{1}{C}} \right)}{{I_{\max}\left( {1 + \frac{1}{C}} \right)} + {2I_{back}}}$

where Vis the visibility, I_(max) is the brightness of the virtual imagelight as indicated by the image information, I_(back) is related to alight value associated with the world object as indicated by the lightinformation (which may be modified by the determined dimming value), andC is a desired contrast (e.g., 100:1). For example, the visibilityequation may be used at each pixel location of the dimmer to calculate adimming value for the particular pixel location using the brightness ofthe virtual image light at the particular pixel location and the lightvalue associated with the world object at the particular pixel location.In some embodiments, I_(back) may be defined using the followingequation:

I _(back) =T _(v) *I _(world)

where T_(v) is the percentage of light that is allowed to pass throughone or more pixels of the dimmer, and I_(world) is the brightness ofambient light from the world as indicated by the light information. Insome examples, T_(v) may be representative of or related to a dimmingvalue.

At step 416, the dimmer is adjusted to reduce an intensity of the lightassociated with the object in the portion of the system field of view.For example, the dimmer may be adjusted such that the intensity of thelight associated with the object impinging on each pixel location of thedimmer is reduced according to the dimming value determined for thatparticular pixel location. As used in the present disclosure, adjustingthe dimmer may include initializing the dimmer, activating the dimmer,powering on the dimmer, modifying or changing a previously initialized,activated, and/or powered on dimmer, and the like. In some embodiments,the processor may send data to the dimmer indicating both the portion ofthe system field of view and the plurality of spatially-resolved dimmingvalues.

At step 418, the projector is adjusted to adjust a brightness associatedwith the virtual image light. For example, in some embodiments it may bedifficult to achieve a desired opacity or visibility of the virtualcontent without increasing or decreasing the brightness of the virtualobject. In such embodiments, the brightness of the virtual image lightmay be adjusted before, after, simultaneously, or concurrently withadjusting the dimmer.

FIG. 5 illustrates an AR device 500 with an eyepiece 502 and a pixelateddimming element 503 consisting of a spatial grid of dimming areas (i.e.,pixels) that can have various levels of dimming. Each of the dimmingareas may have an associated size (i.e., width) and an associatedspacing (i.e., pitch). As illustrated, the spatial grid of dimming areasmay include one or more dark pixels 506 providing complete dimming ofincident light and one or more clear pixels 508 providing completetransmission of incident light. Adjacent pixels within pixelated dimmingelement 503 may be bordering (e.g., when the pitch is equal to the size)or may be separated by gaps (e.g., when the pitch is greater than thesize). In various embodiments, pixelated dimming element 503 may employliquid crystal technology such as dye doped or guest host liquidcrystals, twisted nematic (TN) or vertically aligned (VA) liquidcrystals, or ferroelectric liquid crystals. In some embodiments,pixelated dimming element 503 may comprise an electrochromic device,among other possibilities. In some implementations, pixelated dimmingelement 503 may employ electrically controlled birefringence (“ECB”)technology, such as an ECB cell.

FIG. 6 illustrates a technique for determining the gaze vector based onthe pupil position of the eye of the user. In some instances, the pupilposition relative to the AR device is detected using the eye tracker andthe gaze vector is subsequently defined as the vector orthogonal to thesurface of the eye at the pupil position. The gaze vector mayalternatively or additionally be defined as the vector intersecting thecenter of rotation of the eye and the pupil position. The center ofrotation may be estimated using data gathered by the eye tracker. Thegaze vector may alternatively or additionally be defined as the vectorintersecting the geometric center of the eye and the pupil position. Thegeometric center of the eye may be estimated using data gathered by theeye tracker. Other possibilities are contemplated.

One of several inherent problems with using the pupil position todetermine the gaze vector is illustrated in FIG. 6. In the upperdiagram, a first distance D₁ between the pupil position and the eyepieceis shown when the eye is looking generally toward the center of theeyepiece. In the lower diagram, a second distance D₂ between the pupilposition and the eyepiece is shown when the eye is looking generallytoward the top of the eyepiece. Here, the first distance D₁ is less thanthe second distance D₂, causing render registration problems due to thevarying vergence distance as the eye of the user moves.

FIG. 7 illustrates a technique for determining the gaze vector based onthe center of rotation of the eye of a user. This technique is describedin depth in U.S. Application No. 62/618,559, filed on Jan. 17, 2018,titled “EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, ANDRENDER CAMERA POSITIONING IN DISPLAY SYSTEMS”, the disclosure of whichis incorporated by reference herein. The center of rotation may beestimated using data gathered by the eye tracker, and the gaze vectormay subsequently be defined as the vector formed by connecting thecenter of rotation and the pupil position. One of several benefits ofusing the center of rotation for determining the gaze vector is that thedistance between the center of rotation and the eyepiece may be the sameirrespective of the direction the eye is looking. In the upper diagramof FIG. 7, a third distance D₃ between the center of rotation and theeyepiece is shown when the eye is looking generally toward the center ofthe eyepiece. In the lower diagram, a fourth distance D₄ between thecenter of rotation and the eyepiece is shown when the eye is lookinggenerally toward the top of the eyepiece. Here, the third distance D₃ isthe same as the fourth distance D₄ thereby improving the renderregistration.

FIG. 8 illustrates a technique for determining the gaze vector based ondetected light information and the cone and rod locations within theeye. Because cones are more sensitive to light in high light conditionsand rods are more sensitive to light in low light conditions, as thedetected ambient light decreases (e.g., the global light value), theorigin of the gaze vector may be adjusted from a center position of theretinal layer corresponding to a high density of cones outward to one ormore points along an annulus corresponding to a high density of rods.Accordingly, in high light conditions the determined gaze vector may bea single gaze vector formed by connecting a center position of theretinal layer to the pupil position, and in low light conditions thedetermined gaze vector(s) may be a single or a plurality of gaze vectorsformed by connecting one or more points along an annulus surrounding thecenter position of the retinal layer to the pupil position.Alternatively or additionally, the plurality of gaze vectors may bedescribed/represented as a cone of gaze vectors or a “gaze cone”comprising an infinite number of possible gaze vectors.

Cone and rod locations may be estimated using information gathered bythe eye tracker or, in some embodiments, the center position of theretinal layer corresponding to a high density of cones may be defined bycontinuing the gaze vector determined using the pupil position throughthe eye toward the back of the eye such that the gaze vector determinedusing the pupil position is co-linear with the gaze vector determinedusing cone and rod locations in high light conditions. In someembodiments, the AR device is configured such that the gaze vector isdetermined using cone and rod locations in low light conditions (e.g.,“low light mode”) and is determined using the center of rotation of theeye in high light conditions. In such embodiments, a light threshold maybe established that a detected light value may be evaluated against,causing the gaze vector to be determined using cone and rod locationswhen the detected light value is below the light threshold and causingthe gaze vector to be determined using the center of rotation of the eyewhen the detected light value is above the light threshold.

In some embodiments in which the dimmed area is significantly largeand/or the dimming values are significantly high, the detected ambientlight using the light sensor of the AR device may not be indicative ofthe actual amount of light reaching the eye. In such embodiments, thesize of the pupil may be used as a proxy for the amount of lightreaching the eye. For example, the AR device may switch to a “low lightmode” (causing the gaze vector to be determined using cone and rodlocations) when the pupil size exceeds a pupil size threshold. Forexample, in some implementations the pupil size threshold may be set tobe 20% above an average pupil size of a user in high light conditions(e.g., pupil size may correspond to area, diameter, circumference, etc.of the pupil). In another particular embodiment, the pupil sizethreshold may be predetermined based on average known pupil sizes in lowlight and high light conditions. Other possibilities are contemplated.

FIG. 9 illustrates a determined gaze vector in high light conditions inwhich the pupil is contracted. In some embodiments, the pupil size maybe used to estimate the ambient light (e.g., the global light value) or,alternatively or additionally, the origin(s) of the gaze vector(s) maybe determined directly using the pupil size without estimation ordetection of the ambient light. For example, different pupil diametersmay be related to different cone and rod locations within the retinallayer at which the origin(s) of the gaze vector(s) may be defined.

FIG. 10 illustrates determined gaze vectors in low light conditions inwhich the pupil is dilated. Similar to the scenario in high lightconditions, in low light conditions the pupil size may be used toestimate the ambient light (e.g., the global light value) or,alternatively or additionally, the origin(s) of the gaze vector(s) maybe determined directly using the pupil size.

FIG. 11 illustrates three techniques for determining the gaze vector inhigh light conditions and the corresponding dimmed areas determinedusing each of the three techniques. In the first technique, the gazevector is determined using the pupil position, i.e., “GAZE VECTOR(A)”,resulting in a gaze vector that extends orthogonally from the surface ofthe pupil toward the dimmed area A (or in some embodiments, the areathat is not dimmed). In the second technique, the gaze vector isdetermined using cone and rod locations within the eye, i.e., “GAZEVECTOR(B)”, resulting in a gaze vector that extends from a centerposition of the retinal layer through the pupil position toward thedimmed area B (or in some embodiments, the area that is not dimmed). Thesecond technique may be further facilitated by one or more of: the pupilposition (for providing a second point for defining the gaze vector),the detected ambient light (for determining the origin(s) of the gazevector(s) along the retinal layer), and the pupil size/diameter (forestimating the ambient light and/or for directly determining theorigin(s) of the gaze vector(s) along the retinal layer). In the thirdtechnique, the gaze vector is determined using the center of rotation ofthe eye, i.e., “GAZE VECTOR(C)”, resulting in a gaze vector that extendsfrom the center of rotation of the eye through the pupil position towardthe dimmed area C (or in some embodiments, the area that is not dimmed).In the example of FIG. 12, dimmed area A is the same as dimmed area B.

FIG. 12 illustrates the same techniques illustrated in FIG. 11 but inlow light conditions. The determined gaze vectors and the correspondingdimmed areas are the same using the first and third techniques (usingpupil position and center of rotation, respectively) but have beenmodified using the second technique (using cone and rod locations). Inthe second technique, the gaze vectors are determined using cone and rodlocations within the eye, i.e., “GAZE VECTOR(B′)”, resulting in a set ofgaze vectors that extend from various points along an annulussurrounding the center position of the retinal layer through the pupilposition toward the dimmed area B′ (or in some embodiments, the areathat is not dimmed). In the example shown in FIG. 12, each of dimmedareas A, B′, and C differ from each other.

FIG. 13 illustrates a dimmer having been adjusted to produce dimmed areaA determined using a gaze vector calculated using the pupil position.

FIG. 14 illustrates a dimmer having been adjusted to produce dimmed areaB determined using a gaze vector calculated using cone and rod locationsin high light conditions.

FIG. 15 illustrates a dimmer having been adjusted to produce dimmed areaB′ determined using a gaze vector calculated using cone and rodlocations in low light conditions. In alternative embodiments, dimmedarea B′ may include only portions of the annular region shown in FIG. 15and not the region in its entirety.

FIG. 16 illustrates an example in which dimmed area B′ further includesthe center portion within the annular region.

FIG. 17 illustrates a dimmer having been adjusted to produce dimmed areaC determined using a gaze vector calculated using the center of rotationof the eye.

FIGS. 18A and 18B illustrate an approach for determining a portion ofthe system field of view to be dimmed based on image information. Forexample, one or more steps shown in FIGS. 18A and 18B may correspond tosteps 410 and/or 412. In some embodiments, the AR device may projectlight onto the eyepiece in such a way that virtual content is perceivedby the user at various points in space beyond the eyepiece and thedynamic dimmer, such as points 1802. Points 1802 may, for example,correspond to locations in three-dimensional space including locationsat which pixels of virtual content (e.g., one or more virtual objects)are to be perceived by the user when presented through the eyepiece,locations at which dark virtual content (e.g., a virtual “shadow” castby or otherwise associated with virtual content presented through theeyepiece) is to be perceived by the user, locations physically occupiedby one or more real-world objects or persons located in the user'senvironment (e.g., a virtual black “top hat” anchored to the head ofsomeone in the user's environment), and the like. In someimplementations, points 1802 may be randomly sampled from the virtualcontent or, in some embodiments, points 1802 may be selected based onkey features of the virtual content, such as edges, corners, centers ofsurfaces, among other possibilities. In some embodiments, points 1802may be sampled from the outer perimeter of the virtual content (asviewed from a reference point). In other embodiments, or in the sameembodiments, an image brightness of the virtual content is alsodetermined at each of points 1802, which may be used to determine alevel of dimming (i.e., dimming value) at points 1802 to achieve adesired visibility V of the virtual content. The number of points 1802used may vary based on a speed-accuracy tradeoff.

To dim in alignment with the perceived virtual content, vectors 1804 maybe defined as intersecting each of points 1802 and the pupil position(i.e., a reference point). Intersection points 1806 may then be definedat each location where vectors 1804 intersect with the dynamic dimmer.As shown in reference to FIG. 18B, dimmed portions 1808 may bedetermined based on intersection points 1806. In some implementations,one or more ray- or cone-casting techniques may be employed to definevectors 1804 and identify or otherwise determine intersection points1806. In some embodiments, each of dimmed portions 1808 may be set to anarea encompassing each of intersection points 1806, or to particularpixels of the dynamic dimmer encompassing intersection points 1806. Insome embodiments, the size of dimmed portions 1808 may be a function ofthe number of sampled points 1802 and/or the density of points 1802. Forexample, in some instances the size of dimmed portions 1808 may beinversely proportional to the number of points 1802. In embodiments inwhich points 1802 are sampled from the outer perimeter of the virtualcontent, dimmed portions 1808 may be formed by connecting neighboringintersection points 1806 and dimming the enclosed area. In someexamples, the size and/or shading of dimmed portions 1808 may be afunction of determined distances from the reference point tointersection points 1806, determined distances from intersection points1806 to points 1802, or a combination thereof. In the example of FIGS.18A and 18B, the pupil position (e.g., center of the pupil), which isthe location from which vectors 1804 are defined (i.e., a referencepoint), may change over time as eye movement occurs. As such, thelocations of intersection points 1806 and dimmed portions 1808 may alsochange over time as eye movement occurs.

FIGS. 19A and 19B illustrate an approach for determining a portion ofthe system field of view to be dimmed based on image information similarto shown in reference to FIGS. 18A and 18B but with a differentreference point. Points 1902 may represent different points in spacewhere the virtual content is perceived by the user. Vectors 1904 may bedefined as intersecting each of points 1902 and the center of rotationof the eye (i.e., a reference point). Intersection points 1906 may thenbe defined at each location where vectors 1904 intersect with thedynamic dimmer. As shown in reference to FIG. 19B, dimmed portions 1908may be determined based on intersection points 1906. In someembodiments, each of dimmed portions 1908 may be set to an areaencompassing each of intersection points 1906 or to particular pixels ofthe dynamic dimmer encompassing intersection points 1906. In someexamples, the size and/or shading of dimmed portions 1908 may be afunction of determined distances from the reference point tointersection points 1906, determined distances from intersection points1906 to points 1902, or a combination thereof. The position of thecenter of rotation of the eye, which is the location from which vectors1904 are defined (i.e., a reference point) in the example of FIGS. 19Aand 19B, may be more stable over time as eye movement occurs than thatof the pupil position, which is the reference point in the example ofFIGS. 18A and 18B. It follows that, in the example of FIGS. 19A and 19B,the locations of intersection points 1906 and dimmed portions 1908 mayremain static or change relatively little over time as eye movementoccurs. Although the pupil position and the center of rotation of theeye are described above in reference to FIGS. 18A, 18B, 19A, and 19B asexamples of reference points that may be utilized in determining aportion of the system field of view to be dimmed, it is to be understoodthat examples of such reference points may also include any of a varietyof other locations along the optical axis of the eye. Systems andtechniques for identifying the optical axis of the eye and the locationsof particular anatomical regions of the eye that lie along the opticalaxis, such as the center of the pupil and the center of rotation of theeye, are described in further detail in U.S. Application No. 62/618,559,filed on Jan. 17, 2018, titled “EYE CENTER OF ROTATION DETERMINATION,DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAYSYSTEMS,” which, as mentioned above, is incorporated by reference hereinin its entirety.

FIG. 20 illustrates examples of improving the solidity of displayedvirtual content using any of the techniques described herein, such asadjusting the dimmer and/or adjusting the projector based on lightinformation, gaze information, and/or image information. In reference toboth the left and right side field of views, virtual content 2004 whichis displayed alongside world objects 2002 appears washed out except forportions 2006 of virtual content 2004 where the virtual content appearsmore solid than the remaining portions of virtual content 2004. As shownin the illustrated examples, the solidity of the virtual content isimproved only at the portions of the system field of view where the useris looking.

FIG. 21 illustrates an example of improving the solidity of a displayedvirtual object 2102 by dimming a portion of the system field of viewcorresponding to the virtual object. As illustrated, the opacity andvisibility of a portion 2104 of virtual object 2102 in the region withdimming is relatively greater than that of a portion 2106 of virtualobject 2102 in the region without dimming. By dimming the lightassociated with world objects 2108 at portion 2104, the virtual contentcan be more clearly perceived by the user.

FIG. 22 illustrates a plot showing the relationship between virtualimage light brightness (x-axis) and ambient light levels for maintaininga visibility equal to 0.7 (i.e., V=0.7). The solid slanted lines arefixed visibility level lines (for V=0.7) for different ambient lightlevel conditions. For example, for a projector brightness of 200 nitsbeing used in an indoor area of about 100 nits, a dimming level close to30% may be employed to keep the visibility close to 0.7. Referring onceagain to the visibility equation described above in reference to FIG. 4,in some examples, the x and y axes of the plot illustrated in FIG. 22may correspond to I_(max) and T_(v), respectively, while the solidslanted lines are fixed visibility level lines (for V=0.7) for differentI_(world) values.

FIGS. 23A and 23B illustrate diagrams showing the effect of a smallocclusion on a world scene. FIG. 23A illustrates a simple case in whichan eye of the user is looking at infinity. The eye includes a retina2302, a pupil 2304, and a lens 2306. Light from different angles arefocused to different positions on retina 2302. FIG. 23B shows anocclusor 2308 placed in front of the eye at a distance d away from pupil2304. A gradient disk at the retina may be constructed using simple raygeometry. Ignoring diffraction, the relative transmission at the centerof the gradient disk is t₀=1−(h/p)², where h is the diameter of theocclusor and p is the diameter of the pupil. Put another way, to=1−A_(occlusor)/A_(pupil), where A_(ocausor) is the area of the occlusorand A_(pupil) is the area of the pupil.

FIG. 24 illustrates a plot showing the effect of varying the occlusordiameter on the transmission of the dimming element as a function ofangular extent (in degrees). As illustrated, a smaller occlusor diameter(e.g., 1 mm) has very little effect on the transmission but is much morestable over angular extent than a larger occlusor diameter (e.g., 4 mm)which has a higher effect on the transmission which varies significantlyover angular extent.

FIG. 25 illustrates an example of dimming using a single occlusor inwhich d=17 mm, p=4 mm, and h=1 mm. The dimmed area shows a point spreadfunction (PSF) 2502 of a single pixel. Using the dimming shown, thepixel size requirement for the particular dimming element used can beestimated as a 200 μm pixel.

FIG. 26 illustrates an example of an architecture of an OST head-mounteddisplay (HMD) consisting of a diffractive waveguide eyepiece 2602 thatdelivers the virtual content to the user's eyes. The diffractivewaveguide eyepiece 2602 may include one or more diffractive opticalelements (DOEs), such as an in-coupling grating (ICG), an orthogonalpupil expander (OPE), and/or an exit pupil expander (EPE). The worldlight also passes through the same element to reach the user's eyes. Asshown, a dynamic dimmer 2604 allows management of the world light levelto keep the virtual content at a certain opacity level. In someembodiments, the dimmer 2604 may correspond to a pixelated dimmingelement that is functionally similar or equivalent to pixelated dimmingelement 503 as described above in reference to FIG. 5. In otherembodiments, the dimmer 2604 may correspond to a global (non-pixelated)dimming element. As shown in FIG. 26, in some implementations, thedimmer 2604 may be shaped and curved independent of the eyepiece so asto improve the aesthetics and/or the functionality of the OST-HMD.

FIG. 27 illustrates an additional example of an architecture of anOST-HMD consisting of a micro-display (e.g., LCOS, MEMS, or fiberscanner display type) that delivers light with a relay optics systeminto an in-coupling grating of a diffractive waveguide structure. Thewaveguide structure may include an outcoupling grating (e.g., EPE) thatmagnifies the input image plane and delivers to the user's eyebox. Asshown, various elements may be positioned between the user's eye and theworld objects. An eyepiece 2702 may be a diffractive waveguide combinerthat delivers the virtual light to the user's eye and also allows theworld light to transmit through. A variable focal element 2704 mayconsist of a depth plane variation/switching element between the eye andthe eyepiece to act on the virtual display. In some embodiments,variable focus element 2704 is a back lens assembly (BLA) 2706. The BLAalso invariably acts on the world light and therefore a front lensassembly (FLA) 2708 is added to cancel the impact on the world display.

A dynamic dimming element 2710 in this embodiment is mounted on theoutside of the integrated stack. This allows switching from atransparent display for an AR mode to an opaque display completelyblocking out the world light for a VR mode. The dimming element 2710 maycorrespond to a global dimming element or a pixelated dimming element.An external lens 2712 is positioned separate from the optical stack soas to provide a protective and/or supportive structure for the OST-HMD.External lens 2712 may also provide an amount of dimming to the entiresystem field of view.

FIG. 28 illustrates an additional example of an architecture of anOST-HMD in which a flat dynamic dimmer 2802 is positioned along theinside of a curved external cosmetic lens 2804. The dimmer 2802 maycorrespond to a global dimming element or a pixelated dimming element.In some embodiments, external cosmetic lens 2804 may provide an amountof dimming to the entire system field of view which may be accounted forwhen determining the spatially-resolved dimming values of the dynamicdimmer. The OST-HMD may also include an eyepiece 2806, an adaptive BLA2808, and an adaptive FLA 2810, as described herein.

FIG. 29 illustrates a schematic view of an AR device 2900 according tothe present invention. AR device 2900 generally includes a local module2910 and a remote module 2912. Partitioning of components of AR device2900 between local module 2910 and remote module may allow separation ofbulky and/or high power-consuming components from those positioned closeto the user's head when AR device 2900 is in use, thereby increasinguser comfort as well as device performance. Local module 2910 may behead mounted and may include various mechanical and electronic modulesto facilitate control of pixelated dimmers 2903 and spatial lightmodulators 2904. Control of spatial light modulators 2904 may causevirtual content to be projected onto eyepieces 2902 which, inconjunction with world light modified by dimmers 2903, are viewed by theuser of AR device 2900. One or more sensors 2934 of local module 2910may detect information from the world and/or the user and send thedetected information to a sensor headset processor 2940 which may send adata stream to a display headset processor 2942 of local module 2910 andraw or processed images to perception processing unit 2944 of remotemodule 2912.

In some embodiments, one or more components of local module 2910 may besimilar to one or more components described with reference to FIG. 3.For example, in such embodiments, the functionality of eyepieces 2902and dimmers 2903 may be similar to that of eyepieces 302 and dimmers303, respectively. In some examples, the one or more sensors 2934 mayinclude one or more world cameras, ambient light sensors, and/or eyetrackers similar to one or more of world cameras 306, ambient lightsensor 334, and/or eye trackers 340, respectively. In some embodiments,the functionality of spatial light modulators 2904 may be similar tothat of one or more components included in projectors 314, and thefunctionality of one or both of the sensor headset processor 2940 andthe display headset processor 2942 may be similar to that of one or morecomponents included in processing module 350.

In some embodiments, display headset processor 2942 may receive virtualcontent data and pixelated dimmer data from a graphics processing unit(GPU) 2946 of remote module 2912 and may perform various correction andwarping techniques prior to controlling pixelated dimmers 2903 andspatial light modulators 2904. Dimmer data generated by display headsetprocessor 2942 may pass through one or more drivers which may modify orgenerate voltages for controlling dimmers 2903. In some embodiments,display headset processor 2942 may receive a depth image and a headsetpose from sensor headset processor 2940 which can be used to improve theaccuracy of the dimming and the projected virtual content.

Remote module 2912 may be electrically coupled to local module 2910through one or more wired or wireless connections, and may be fixedlyattached to the user or carried by the user, among other possibilities.Remote module 2912 may include a perception processing unit 2944 forperforming/generating an environment lighting map, a headset pose, andeye sensing.

Perception processing unit 2944 may send data to a CPU 2948 which may beconfigured to perform/generate passable world geometry and app scenegeometry. CPU 2948 may send data to GPU 2946 which may be configured toperform a check for a minimum world luminance throughput, a dimmingpixel alignment, a late frame time warp, and a render pipeline, amongother operations. In some embodiments, CPU 2948 may be integrated withGPU 2946 such that a single processing unit may perform one or more ofthe functions described in reference to each. In some embodiments, thefunctionality of one or more of the components included in remote module2912 may be similar to that of one or more components included inprocessing module 350.

FIG. 30 illustrates a method 3000 for sharpening out-of-focus pixelateddimming. Method 3000 may be used in addition to method 400 to improvethe performance of the dimmer. For example, one or more steps of method3000 may be performed prior to step 416 and/or subsequent to step 414.Steps of method 3000 may be performed in a different order than thatshown in FIG. 30, and not all of the steps need be performed. Forexample, in some embodiments, one or more of steps 3002, 3004, and 3006may be omitted during performance of method 3000. One or more steps ofmethod 3000 may be performed by a processor (e.g., processor 352) or bysome other component within the AR device.

At step 3002, a pixelated mask is generated. When method 3000 is used inconjunction with method 400, the pixelated mask may be generated basedon the portion of the system field of view to be dimmed determined instep 412 and/or the dimming values determined in step 414 (i.e., dimmedarea 236). In some embodiments, the pixelated mask may be generatedusing one or both of the techniques described in reference to FIGS. 31and 32.

At step 3004, the dimmer is adjusted in accordance with the pixelatedmask. For example, each pixel of the dimmer may be set to a particulardimming value as indicated by the pixelated mask. For examples in whichmethod 3000 is used in conjunction with method 400, one or moreoperations of step 3004 may at least in part correspond to one or moreoperations of step 416.

At step 3006, the user looks through the dimmer and the observabledimming (to the user) is equivalent to the pixelated mask convolved withthe PSF of a single pixel. Accordingly, step 3006 is inherentlyperformed by the optics of the eye when the user is wearing the ARdevice, rather than being performed directly by a component of the ARdevice. On the other hand, steps 3002 and 3004 may, for example, beperformed by one or more components of the AR device.

FIG. 31 illustrates a method 3100 for sharpening out-of-focus pixelateddimming in which step 3102 comprises step 3108 at which a desiredpixelated mask is generated and step 3110 at which the pixelated mask isset to the desired pixelated mask. The observable dimming to the user(shown adjacent to step 3106) is significantly blurred in comparison tothe pixelated mask (shown adjacent to step 3104) due to the smearingcaused by the PSF of a single pixel. In some embodiments, one or moresteps of method 3100 may correspond to one or more steps of method 3000.

FIG. 32 illustrates a method 3200 for sharpening out-of-focus pixelateddimming in which step 3202 includes a deconvolution technique. At step3208, the diameter of the pupil of the eye of the user is detected usinga sensor directed at the eye (e.g., a camera such as eye tracker 240).At step 3210, the accommodative state of the eye lens is detected usingthe same or a different sensor as used in step 3208.

At step 3212, the PSF of a single pixel is estimated based on the pupildiameter, the accommodative state of the eye lens, the size/shape of apixel, and the distance from a pixel to the pupil. Where the shape of apixel can be approximated using a circle, the size/shape of a pixel maybe represented as diameter h, the pupil diameter as diameter p, and thedistance from a pixel to the pupil as distance d (using the nomenclatureestablished in FIGS. 23A and 23B). In some embodiments, the distancefrom a pixel to the pupil may be different for different pixels of thedimmer such that the estimated PSF may be pixel dependent. In someembodiments, the distance from the centermost pixel to the pupil is usedas an approximation for the remaining pixels.

At step 3214, a desired pixelated mask at the depth of the virtualcontent is generated. At step 3216, the desired pixelated mask isdeconvolved using the estimated PSF of a single pixel. In someembodiments, the deconvolution is performed in the spatial frequencydomain by dividing the Fourier Transform of the desired pixelated maskat the depth of the virtual content by the Fourier Transform of theestimated PSF of a single pixel and performing an inverse-FourierTransform. Alternatively or additionally, the deconvolution may beperformed dividing the Fourier Transform of the estimated PSF of asingle pixel by the Fourier Transform of the desired pixelated mask atthe depth of the virtual content and performing an inverse-FourierTransform. The pixelated mask used subsequently in step 3204 is set tothe result of the deconvolution.

As a result of performing method 3200, the observable dimming to theuser (shown adjacent to step 3206) is significantly less blurred incomparison to the technique in method 3100 despite the dissimilaritybetween the pixelated mask (shown adjacent to step 3204) and the desiredpixelated mask (shown adjacent to step 3214). In some embodiments, oneor more steps of method 3200 may correspond to one or more steps ofmethods 3000 and 3100. In some embodiments, one or more steps of method3200 may be omitted or modified.

FIG. 33 illustrates a simplified computer system 3300 according to someembodiments described herein. Computer system 3300 as illustrated inFIG. 33 may be incorporated into devices such as AR device 200 or 300 asdescribed herein. FIG. 33 provides a schematic illustration of oneexample of computer system 3300 that can perform some or all of thesteps of the methods provided by various embodiments. It should be notedthat FIG. 33 is meant only to provide a generalized illustration ofvarious components, any or all of which may be utilized as appropriate.FIG. 33, therefore, broadly illustrates how individual system elementsmay be implemented in a relatively separated or relatively moreintegrated manner.

Computer system 3300 is shown comprising hardware elements that can beelectrically coupled via a bus 3305, or may otherwise be incommunication, as appropriate. The hardware elements may include one ormore processors 3310, including without limitation one or moregeneral-purpose processors and/or one or more special-purpose processorssuch as digital signal processing chips, graphics accelerationprocessors, and/or the like; one or more input devices 3315, which caninclude without limitation a mouse, a keyboard, a camera, and/or thelike; and one or more output devices 3320, which can include withoutlimitation a display device, a printer, and/or the like.

Computer system 3300 may further include and/or be in communication withone or more non-transitory storage devices 3325, which can comprise,without limitation, local and/or network accessible storage, and/or caninclude, without limitation, a disk drive, a drive array, an opticalstorage device, a solid-state storage device, such as a random accessmemory (“RAM”), and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable, and/or the like. Such storage devicesmay be configured to implement any appropriate data stores, includingwithout limitation, various file systems, database structures, and/orthe like.

Computer system 3300 might also include a communications subsystem 3319,which can include without limitation a modem, a network card (wirelessor wired), an infrared communication device, a wireless communicationdevice, and/or a chipset such as a Bluetooth™ device, an 802.11 device,a WiFi device, a WiMax device, cellular communication facilities, etc.,and/or the like. The communications subsystem 3319 may include one ormore input and/or output communication interfaces to permit data to beexchanged with a network such as the network described below to name oneexample, other computer systems, television, and/or any other devicesdescribed herein. Depending on the desired functionality and/or otherimplementation concerns, a portable electronic device or similar devicemay communicate image and/or other information via the communicationssubsystem 3319. In other embodiments, a portable electronic device, e.g.the first electronic device, may be incorporated into computer system3300, e.g., an electronic device as an input device 3315. In someembodiments, computer system 3300 will further comprise a working memory3335, which can include a RAM or ROM device, as described above.

Computer system 3300 also can include software elements, shown as beingcurrently located within the working memory 3335, including an operatingsystem 3340, device drivers, executable libraries, and/or other code,such as one or more application programs 3345, which may comprisecomputer programs provided by various embodiments, and/or may bedesigned to implement methods, and/or configure systems, provided byother embodiments, as described herein. Merely by way of example, one ormore procedures described with respect to the methods discussed above,might be implemented as code and/or instructions executable by acomputer and/or a processor within a computer; in an aspect, then, suchcode and/or instructions can be used to configure and/or adapt a generalpurpose computer or other device to perform one or more operations inaccordance with the described methods.

A set of these instructions and/or code may be stored on anon-transitory computer-readable storage medium, such as the storagedevice(s) 3325 described above. In some cases, the storage medium mightbe incorporated within a computer system, such as computer system 3300.In other embodiments, the storage medium might be separate from acomputer system e.g., a removable medium, such as a compact disc, and/orprovided in an installation package, such that the storage medium can beused to program, configure, and/or adapt a general purpose computer withthe instructions/code stored thereon. These instructions might take theform of executable code, which is executable by computer system 3300and/or might take the form of source and/or installable code, which,upon compilation and/or installation on computer system 3300 e.g., usingany of a variety of generally available compilers, installationprograms, compression/decompression utilities, etc., then takes the formof executable code.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware might also be used, and/or particularelements might be implemented in hardware, software including portablesoftware, such as applets, etc., or both. Further, connection to othercomputing devices such as network input/output devices may be employed.

As mentioned above, in one aspect, some embodiments may employ acomputer system such as computer system 3300 to perform methods inaccordance with various embodiments of the technology. According to aset of embodiments, some or all of the procedures of such methods areperformed by computer system 3300 in response to processor 3310executing one or more sequences of one or more instructions, which mightbe incorporated into the operating system 3340 and/or other code, suchas an application program 3345, contained in the working memory 3335.Such instructions may be read into the working memory 3335 from anothercomputer-readable medium, such as one or more of the storage device(s)3325. Merely by way of example, execution of the sequences ofinstructions contained in the working memory 3335 might cause theprocessor(s) 3310 to perform one or more procedures of the methodsdescribed herein. Additionally or alternatively, portions of the methodsdescribed herein may be executed through specialized hardware.

The terms “machine-readable medium” and “computer-readable medium,” asused herein, refer to any medium that participates in providing datathat causes a machine to operate in a specific fashion. In embodimentsimplemented using computer system 3300, various computer-readable mediamight be involved in providing instructions/code to processor(s) 3310for execution and/or might be used to store and/or carry suchinstructions/code. In many implementations, a computer-readable mediumis a physical and/or tangible storage medium. Such a medium may take theform of a non-volatile media or volatile media. Non-volatile mediainclude, for example, optical and/or magnetic disks, such as the storagedevice(s) 3325. Volatile media include, without limitation, dynamicmemory, such as the working memory 3335.

Common forms of physical and/or tangible computer-readable mediainclude, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, punchcards, papertape, any other physical medium with patternsof holes, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip orcartridge, or any other medium from which a computer can readinstructions and/or code.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to the processor(s) 3310for execution. Merely by way of example, the instructions may initiallybe carried on a magnetic disk and/or optical disc of a remote computer.A remote computer might load the instructions into its dynamic memoryand send the instructions as signals over a transmission medium to bereceived and/or executed by computer system 3300.

The communications subsystem 3319 and/or components thereof generallywill receive signals, and the bus 3305 then might carry the signalsand/or the data, instructions, etc. carried by the signals to theworking memory 3335, from which the processor(s) 3310 retrieves andexecutes the instructions. The instructions received by the workingmemory 3335 may optionally be stored on a non-transitory storage device3325 either before or after execution by the processor(s) 3310.

The methods, systems, and devices discussed above are examples. Variousconfigurations may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods may be performed in an order different from that described,and/or various stages may be added, omitted, and/or combined. Also,features described with respect to certain configurations may becombined in various other configurations. Different aspects and elementsof the configurations may be combined in a similar manner. Also,technology evolves and, thus, many of the elements are examples and donot limit the scope of the disclosure or claims.

Specific details are given in the description to provide a thoroughunderstanding of exemplary configurations including implementations.However, configurations may be practiced without these specific details.For example, well-known circuits, processes, algorithms, structures, andtechniques have been shown without unnecessary detail in order to avoidobscuring the configurations. This description provides exampleconfigurations only, and does not limit the scope, applicability, orconfigurations of the claims. Rather, the preceding description of theconfigurations will provide those skilled in the art with an enablingdescription for implementing described techniques. Various changes maybe made in the function and arrangement of elements without departingfrom the spirit or scope of the disclosure.

Also, configurations may be described as a process which is depicted asa schematic flowchart or block diagram. Although each may describe theoperations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be rearranged. A process may have additional steps notincluded in the figure. Furthermore, examples of the methods may beimplemented by hardware, software, firmware, middleware, microcode,hardware description languages, or any combination thereof. Whenimplemented in software, firmware, middleware, or microcode, the programcode or code segments to perform the necessary tasks may be stored in anon-transitory computer-readable medium such as a storage medium.Processors may perform the described tasks.

Having described several example configurations, various modifications,alternative constructions, and equivalents may be used without departingfrom the spirit of the disclosure. For example, the above elements maybe components of a larger system, wherein other rules may takeprecedence over or otherwise modify the application of the technology.Also, a number of steps may be undertaken before, during, or after theabove elements are considered. Accordingly, the above description doesnot bind the scope of the claims.

As used herein and in the appended claims, the singular forms “a”, “an”,and “the” include plural references unless the context clearly dictatesotherwise. Thus, for example, reference to “a user” includes a pluralityof such users, and reference to “the processor” includes reference toone or more processors and equivalents thereof known to those skilled inthe art, and so forth.

Also, the words “comprise”, “comprising”, “contains”, “containing”,“include”, “including”, and “includes”, when used in this specificationand in the following claims, are intended to specify the presence ofstated features, integers, components, or steps, but they do notpreclude the presence or addition of one or more other features,integers, components, steps, acts, or groups.

It is also understood that the examples and embodiments described hereinare for illustrative purposes only and that various modifications orchanges in light thereof will be suggested to persons skilled in the artand are to be included within the spirit and purview of this applicationand scope of the appended claims.

What is claimed is:
 1. A method of operating an optical system, the method comprising: receiving, at the optical system, light associated with a world object; projecting virtual image light onto an eyepiece; determining a portion of a system field of view of the optical system to be at least partially dimmed based on information detected by the optical system; and adjusting a dimmer to reduce an intensity of the light associated with the world object in the portion of the system field of view.
 2. The method of claim 1, wherein the optical system comprises a light sensor configured to detect light information corresponding to the light associated with the world object, wherein the detected information includes the light information.
 3. The method of claim 2, wherein the light information includes a plurality of spatially-resolved light values.
 4. The method of claim 1, wherein the optical system comprises an eye tracker configured to detect gaze information corresponding to an eye of a user of the optical system, wherein the detected information includes the gaze information.
 5. The method of claim 1, further comprising: detecting image information corresponding to the virtual image light, wherein the detected information includes the image information.
 6. The method of claim 5, wherein the image information includes a plurality of spatially-resolved image brightness values.
 7. The method of claim 1, further comprising: determining a plurality of spatially-resolved dimming values for the portion of the system field of view based on the detected information, wherein the dimmer is adjusted according to the plurality of dimming values.
 8. The method of claim 1, wherein the dimmer comprises a plurality of pixels.
 9. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving, at an optical system, light associated with a world object; projecting virtual image light onto an eyepiece; determining a portion of a system field of view of the optical system to be at least partially dimmed based on information detected by the optical system; and adjusting a dimmer to reduce an intensity of the light associated with the world object in the portion of the system field of view.
 10. The non-transitory computer-readable medium of claim 9, wherein the optical system comprises a light sensor configured to detect light information corresponding to the light associated with the world object, wherein the detected information includes the light information.
 11. The non-transitory computer-readable medium of claim 10, wherein the light information includes a plurality of spatially-resolved light values.
 12. The non-transitory computer-readable medium of claim 9, wherein the optical system comprises an eye tracker configured to detect gaze information corresponding to an eye of a user of the optical system, wherein the detected information includes the gaze information.
 13. The non-transitory computer-readable medium of claim 9, wherein the operations further comprise: detecting image information corresponding to the virtual image light, wherein the detected information includes the image information.
 14. The non-transitory computer-readable medium of claim 13, wherein the image information includes a plurality of spatially-resolved image brightness values.
 15. The non-transitory computer-readable medium of claim 9, wherein the operations further comprise: determining a plurality of spatially-resolved dimming values for the portion of the system field of view based on the detected information, wherein the dimmer is adjusted according to the plurality of dimming values.
 16. The non-transitory computer-readable medium of claim 9, wherein the dimmer comprises a plurality of pixels.
 17. An optical system comprising: a projector configured to project virtual image light onto an eyepiece; a dimmer configured to dim light associated with a world object; a processor communicatively coupled to the projector and the dimmer, wherein the processor is configured to perform operations comprising: determining a portion of a system field of view of the optical system to be at least partially dimmed based on information detected by the optical system; and adjusting the dimmer to reduce an intensity of the light associated with the world object in the portion of the system field of view.
 18. The optical system of claim 17, further comprising: a light sensor configured to detect light information corresponding to the light associated with the world object, wherein the detected information includes the light information.
 19. The optical system of claim 17, further comprising: an eye tracker configured to detect gaze information corresponding to an eye of a user of the optical system, wherein the detected information includes the gaze information.
 20. The optical system of claim 17, wherein the operations further comprise: detecting image information corresponding to the virtual image light, wherein the detected information includes the image information. 